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Introduction 

This  paper  discusses  various  aspects  of  the  methodology  of  the 
Physicians'   Practice  Cost  Survey  conducted  by  the  National  Opinion 
Research  Center  under  contract  with  the  Health  Care  Financing 
Administration.   The  aim  of  the  study  was  to  collect  data  on  the  allocation 

of  office  time,  office  costs  and  net  and  gross  income  from  a  sample  of 
approximately  4,000  physicians. 

Given  the  size  of  the  study  and  the  nature  of  the  data  to  be  collected, 
a  great  deal  of  attention  must  be  paid  to  the  behavior  and  quality  of 
the  available  strategies  for  obtaining  the  information.  The  first  part 
of  this  paper  is  a  discussion  of  the  requirements  of  the  study  and 
the  relative  efficacy  of  the  three  main  strategies  of  data  collection-- 
personal  interviews,   telephone  interviews  and  mailed  quest ionnaires -- 
for  meeting  these  requirements.  The  second  part  of  the  paper  is  a  detailed 
discussion  of  the  quality  of  survey  measures  of  income  and  expenses  and 
the  ability  of  each  data  collection  technique  to  yield  high  quality  data. 
The  third  section  is  a  brief  discussion  of  the  goals  and  methods  of 
imputation  procedures  for  handling  the  problem  of  missing  responses  to 
particular  questions. 


c 


-2- 


I 

Design  Considerations  for  the  Physicians'   Practices  Cost  Survey 

Given  the  tasks  of  the  Cost  Survey,  what  is  the  best  strategy  for    _  _ 

collecting  the  data?  In  this  section  we  hope  to  answer  this  question.  y 
In  doing  so  we    vzill  probe  a  little  more  deeply  into  what,  exactly, the  tasks 
are    that  the  survey     is      to  accomplish  and  we  will  show  the  implications 
of  some  of  the  more  important  survey  requirements  for  the  choice  of  a  data 
collection  strategy.  The  detailed  topics  to  be  considered  are:    (a)  the  desired 
response  rate  for  the  survey  (the  response  rate  is  the  number  of  physicians 
ultimately  providing  data  divided  by  the  number  elegible  in  the  sample); 
(b)  the  length  of  time  it  would  take  to  complete  the  task  under  different  field 
strategies;    (c)  im.pl ications  of  the  length  of  the  interview  and  of  the  amount 
of  information  requested;    (d)  the  types  of  questions  asked  in  the  survey;  and, 
finally,    (e)  costs.  At  the  end  of  this  section     is  a  summary  of  the 
strengths  and  weaknesses  of  the  three  main  data  collection  strategies 
(mail,   telephone  and  personal  interviewing)  with  special  attention  to  the 
requirements  of  the  Physicians'   Practices  Cost  Survey, 

Several  books  have  been  written  on  the  topic  of  methods  for  administering 
social  surveys.   In  recent  years  there  have  even  been  books  specifically 
devoted  to  phone  surveys   (Blankenship ,   1977)   ,  mail  surveys   (Erdos ,  1970), 
comparisons  of  phone  and  personal  sur^/eys   (Kahn  and  Groves,   1979;  Lucas  and  Adams, 
1977),     and      comparisons  of  all  three  methods   (Dillman,   1978)     to  accompany 
the  supply  of  data  collection  texts  oriented  to  the  topic  of  personal 

interviewing  (e.g.  Moser  and  Kalton,  1972).  The  lesson  from  these  books  is 
that  each  method  has  its  o\<nx  strengths  and  its        weaknesses.     In  a  recent 

review  of  the  literature  on  all  three  methods,  Dillman  concludes: 
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The  most  important  message  of  this  chapter  is  that  the  question  of  which 
method  is  best  cannot  be  answered  in  abstract  terms.  Although  each 
method  has  certain  strengths  and  weaknesses,  they  do  not  apply  equally, 
or  sometimes  at  all,   to  every  survey  situation.  Thus,  until  the  / 
attributes  of  each  method  are  considered  in  relation  to  the  study 
topic,  the  population  to  be  surveyed  and  the  precise  survey  objectives, 
the  question  of  which  is  best  cannot  be  answered, 
p  39 

Therefore,  we  propose  to  undertake  the  detailed  comparison  Dillman  recommends. 

Response  Rate  It  is  not  a  simple  matter  to  obtain  estimates  of  the 
expected  response  rates  for  the  three  survey  methods  when  the  data  are  to- 
be  collected  from  physicians.  This  is  because  most  studies  find  it  necessary  to 
combine  methods  in  the  overall  data  collection  strategy.    In  the  large  personal 
interview  studies     (e.g.   the  National  Ambulatory  Medical  Care  Survey)  there 
were  significant  phone  and  mail  components.  To  interview  physicians  personally 
it  is  essential,   to  control  field  costs,   to  phone  first  to  schedule  an 
appointment  for  the  interview.  Moreover,  experience  has  shown  that  it  is 
usually  necessary  to  speak  to  the  physician  directly  to  schedule  the 
appointment.  The  usual  practice  in  personal  interview  studies  of  physicians 
is       to  also  mail  an  advance  statament  of  the  aims,   sponsorship  and 

endorsement  of  the  survey.   Such  a  procedure  has  been  shown  to  increase  the 
participation  in  general  population  surveys   (Dillman,  Gallegos  and  Frey,1976) 
and  in  physician  samples   (Minor,  Mullan  and  Loft,  1976), 

So,  when  we  evaluate  personal  interviewing,  xje  are  usually  evaluating 
a  study  design  which  includes  all  three  modes  of  data  collection.  A  telephone 
interviewing  design  is  more  appropriately  thought  of  as  a  combined  mail 
and  telephone  design,   since  the  recommenced  procedure  is  to  preface  Che 

telephone  contact  with  an  advance  statement   (as  is  done  in  the  Physicians' 
Costs  Survey),  There  are  some  reports  of  straight  mail  surveys  of  physician 
populations  and  it  is  to  these  reports  that  we  now  turn. 
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Barnoon  (1974)  reports  that  a  1973  survey  of  laboratory  practices  of 
office  based  physicians  sponsored  by  the  American  Society  of  Internal 
Medicine  achieved  a  response  rate  of  41  to  59  percent  depending  on  the 
specialty.  The  Center  for  Disease  Control   (1974)  reports  that  a  1973  study  / 
of  35,000  physicians  yielded  a  49  percent  response  rate.Mendenhall  (1978) 
reports  that  a  study  of  clinical  practices  sponsored  by  the  University  of 
Southern  California  Division  of  Research  in  Medical  Education  achieved 
a  response  rate  of  53  to  82  percent,  depending  on  the  specialty.  Goodman 
and  Jensen  (1979)  report  that  the  AMA  Periodic  Survey  of  Physicians  generally 
obtains  responses  from  50  percent  of  the  physicians  surveyed.   Finally,  Shosteck 
and  Fairweather  (1979)  report  the  highest  response  rate  for  a  mail  survey 
of  office  based  physicians--65  percent  for  an  FDA  sponsored  survey.  They 
note,  however,  that  it  took  9  weeks  to  obtain  the  192  responses  which  comprise 
the  65  percent  response  rate  in  their  study  comparing  mail  and  personal  interviews. 

These  response  rates  are  lower  than  the  response  rates  achieved  in 
the  mixed  personal/telephone  surveys  and  the  literature  shows  that,  in 
general,   for  any  population  response  rates  are  lower  for  mail  surveys  V7hen 
compared  with  any  other  mixture  of  methods   (Dillman,   1978).  We  are  told  that 
one  of  the  reasons  for  the  low  response  of  physicians  to  mail  surveys  is 
that  they     are  overburdened  with  this  kind  of  paperwork.    It  is  not  uncommon 
for  a  physician  to  receive  60  mail  questionnaires     a  month  from  market 
research  firms.   In  addition,   there  are  a  great  many  government  and  insurance 
forms  that  physicians  regularly  fill  out  that  often  seek  the  same  information 
as  is  requested  .in  mail  questionnaires.    It  is  easy  for  mailed  forms  to 
become  lost  in  the  shuffle  or  to  be  given  to  someone  else,  not  necessarily 
a  knowledgeable  respondent , for  attention. 


-5- 


The  best  example  of  a  personal  inteirview  Ccon±)ined  with  telephone  and 
mail  contact)  data  collection  effort  to  compare  to  the     Physicians  Practices 
Cost  Survey  is  the  National  Ambulatory  Medical  Care  Survey,  conducted  by 
NORC  beginning  in  April,   1973.  This  study  achieved  a  response  rate  of  76,4 
percent  in  the  first  wave  and  is  up  to  81.4  percent  in  the  m.ost  recent 
wave  (Loft,   1979).  This  study  uses  advance  mail  contact  and  phone  appointments 
to  set  up  a  brief  personal  interview.  The  interview  consists  of  10  minutes 
of  data  collection    for  background  information  and  10  minutes  for  explaining 
the  sel f -administered  data  collection  form.s  which  are  dropped  off  at  the 
physician's  office  to  be  filled  out  during  the  next  week.  The  interviewers 
process  about  60  cases  per  week  and  NORC ' s  experience  is  that  it  is 
necessary  to  allow  3  weeks  between  the  time  a  case  enters  the  sample  and 
the  interview  is  completed  because  of  the  difficulties  involved  in  contacting 
the  physician  or  a  knowledgable  representative,   scheduling  an  appointment 
for  an  interview,  and  then  actually  seeing  that  the  interview  appointment  is 
kept  rather  than  delayed,   rescheduled,   refused,  and  so  on, 

70  percent  of  the  survey  refusals  occur  during  the  telephone  contact. 
Of  the  remaining  refusals,   about  half  occur  during  the  induction  interview 
and  half  occur  because  the  physician  will  not  fill  out  the  forms  which 
were  dropped  off  in  the  office  (Loft,   1979).   The  predominant  reason 
given  for  a  refusal  is  that  the  physician  is  too  busy   (52,2  percent). 
The  second  m.ost  important     reason  was  that  the  physician  was  hostile  to 
the  idea  of  surveys  in  general  or  this  survey  in  particular  (25.6  percent), 
3.5  percent  of  the  refusals  occurred  because  the  interviewer  could  not 
get  past  the  receptionist  or  secretary  to  schedule  an  appointm»ent  with  the 
physician.  The  remaining  refusals  were  for  personal'^  reasons   (illness,  death 
of  a  relative--2, 6  percent),       for  other  reasons  not     easily  classified, 
but  mainly  for  multiple  reasons   (e,g.   "I  don't  like  government  sur-zeys  and 
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I'm  too  busy  anywa)^' )  which  could  possibly  be  classified  into  the  first 

or  second  category.  An  analysis  using  AMA.  records     (Loft,   1979)  found  that 

nonrespondents  were  more  likely  to  be  general  or  family  practitioners   (as  opposed 

to  specialists)  and  were  much  more  likely  to  be  older  (over  60  years  of  age) 

and  therefore  many  more  years  out  of  medical  school  than  the  average  respondent. 
The  major  differences  between  the  National  Ambulatory  Medical  Care  Survey 

and  the  Costs  Survey,  as  far  as  the  field  operation  is  concerned  are  that  the  first 

Cost  Surveys  were      preceded  by  a  different  advance  letter  with  no 

endorsements,  the  interview  is  longer  with  m.ore  complex  questions,  there 

are  no  drop  off  data  forms,  and  the  inten/iewing  is  done  by  phone.  Each 

of  these  factors  can  make  a  difference  in  the  response  rate.   For  instance, 

a  current  NORC  study  of  Pediatricians  uses  the  same  design  as  the  National 

Ambulatory  Medical  Care  Survey  with  a  longer  personal  interview.  The  response 

rate  for  this  study  is  near  90  percent,    in  part  (we  believe)  because 

the  advance  letters  indicated  survey  sponsorship  by  the  American  Academy 

of  Pediatrics. 

The  effect  of  the  differences  between  the  National  Am.bulatory  Medical 
Care  Survey  and  the  Costs  Survey  is  to  produce  a  lower  response  rate.   In  the 
first  fielding  of  the  sur/ey,   about  70  percent  of  the  physicians  contacted 
provided  responses.    It  would  be  quite  difficult  to  allocate  the  difference 
in  response  rate  due  to  each  difference  between  the  surveys.  We  note,  however, 
that  the  pattern  in  response  rates  reviewed  here  for  physician  studies  is  the 
sam.e  as  that  obtained  by  Dillman  in  his     discussion  of  the  relative  merits 
of  the  three  methods  of  obtaining  data  from  a  wide  variety  of  populations: 
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The  results  of  our  studies     lead  us  to  conclude  that  response  rates  to 
mail  questionnaires  are  usually  lower  than  those  obtained  by  either 
of  the  interview  methods,  assuming  that  intensive  call-back  efforts  are 
made  for  the  latter,    , . . 

(When  we  ask) .. .what  is  the  likelihood  that  someone  will  agree 
to  be  surveyed  when  actually  confronted  with  the  request,   regardless  of  the 
effort  necessary  to  reach  them.., we  conclude  that  the  face-to-face  interview 
is  best,     the  telephone  a  close  second,  and  the  mail  survey  a  smoewhat 
more  distant  third, 

pp51-52 


Timing  of  the  field  period    We  have  already  noted  that,  compared  to 
other  populations,   there  are  unique  problems  with  scheduling  personal 
interviews  with  physicians  and  maintaining     that  schedule.  The  Shosteck  and 
Fairweather  survey,  described  earlier,  took  15  weeks  to     (_  Ct-^^  fct'^-   /j  <,;>i^< 

physicians   (for  a  response  rate  of  53  percent) .   It  took  this 
long  because  they  had,   in  fact,  underestimated  the  time  that  was  required  to 
contact  physicians  and  fit  interviews  into  physicians'   schedules  (Shosteck 
and  Fairweather,   1979).  The  experience  in  the  Physicians'   Practice  Cost 
Survey  is  that  a  total  of  about  6  hours  is  required  to  contact  and  interview 
the  average  physician  respondent.   The  advantage  of  using  a  telephone 
interview  is  that  the  work  time  can  be  scheduled  and  used  more  efficiently. 
This  observation  es  consistent  with  Dillman's  general  findings: 


As  a  general  rule,   telephone  surveys  are  the  fastest  to  complete. 
Mail  surveys  . , ,   are  usually     locked  into  a  definite  time  schedule  of 
printing,   folding,   stuffing  and  mailing  dates.   Follow-ups  are  also  rigidly 
scheduled,    , . . 

The  time  required  for  implementing  face-to-face  inten/iew  studies  varies 
g  reatly*    The  factors  of  sample  size,   geographic  dispersion  and  effort  required 
to  recruit  and  train  interviewers  suggest  that  the  time  required  is  generally 
far  closer  to  that  required  for  mail  surveys  than  telephone  surveys.  In 
any  case,   the  major  factors  in  such  a  determination  are  the  sample 
size  and  geographical  dispersion. 

pp  67-68  , 
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In  the  case  of  the  Physicians'  Practices  Survey,  there  is     the  added 
consideration  that  telephone  interviewing  is  more  efficient  when  precision  is 
essential  in  timing  the  interviews  and/or  when  the  number  of  callbacks  to 

J 

reschedule  appointments  for  interviews  is  high.   Even  with  these  time 
savings,  we  find  that  the  field  period  for  the  Physicians'   Costs  Survey  is 
22  weeks--long  by  survey  standards  but  far  shorter  than  it  would  be  with 
personal  interviews. 


Length  of  the  Interview      The  interview  for  the  Physicians'  Practice 
Cost  Survey  takes  approximately  45  minutes.    (The  45  minutes  for  the  interview 
should  not  be  confused  with  the  6  hours  required  to  complete  a  case.  The  added 
time  to  complete  the  case  is  due  to  contact  time,   scheduling  time  and  interviewer 
bookesping  time  for  such  matters  as  recording  case  dispositions.)  The  conclusion 
from  Dillman's  study  is  that  the  length  of  the  questionnaire  makes  this 


survey  inappropriate  for  mailing: 

After  response  rate,   the  factor  that  has  cnncributed  most  to  the 
poor  image  of  mail  and  telephone  questionnaires  is  the  viev7  that  they 
must  be  kept  short,   too  short  for  the  study  of  significant  social 
science  questions.  ... 

(Our  analysis  of  mail  questionnaires  finds  that)  there  is  not  much 
difference  in  the  response  rates  for  questionnaires  of  less  than  12 
pages  or  those  which  contain  less  than  125  items  for  either  the 
general  public  or  specialized  populations.  This  suggests  that  increasing 
length  up  to  these  limits  does  not  have  an  adverse  effect  on  response  rates 
It  further  suggests  that  the  maxim  of  "the  shorter  the  better"  may 
represent  an  oversimplification  or  even  a  myth.  However,  going  beyond 
12  pages  seems  almost  certain  to  affect  response,  ... 

In  the  case  of  the  telephone  survey,  Colombotos  reported  no 
difficulty  with  hour-long  interviews  for  a  sample  of  physicians  (1969), 
Telephone  interviews  in  a  leisure  research  study  averaged  20  minutes,  with 
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virtually  no  termination  once  the  interviews  were  underway   (Field,  1973), 
Warland,   in  a  national  survey  on  consumer  behavior  with  interviews  that  lasted 
30  minutes,   found  that  only  4  percent  of  the  sample  terminated  after  the 
interview  had  commenced  (1973).   Studies   .,,  have  shown  that  respondents  . 
terminating  in  the  middle  of  interviews  seldon  exceed  2  percent,  ... 
The  evidence  clearly  suggests  that  once  people  are  on  the  telephone, 
the  length  of  interview  does  not  appear  to  be  a  major  problem. 

pp  54-55  / 

Types  of  Questions  Asked     In  evaluating  the  methodology  of  the  Physicians' 
Practices  Cost     Survey,   there  are  txro  questions  to  be  asked  regarding  the 
kind  of  data  to  be  collected  on  the  survey:    (a)   is  it  possible  that  some 
of  the  questions  to  be  asked  are  too  complicated  or  for  some  other  reason 
technically  inappropriate  for  one  of  the  three  methods  of  data  collection?; 
and  (b)  would  physicians'   answers  to  the  questions  be  different  depending 
on  the  data  collection  method? 

The  Physicians'   Costs  Survey  contains  several  detailed  questions  about 
categories  of  costs  that  might  not  be  uniformly  understood  by  physicians.  In 
addition,   there  are  screening  questions  in  the  survey--i.e.   not  all  of  the 
questions  are  meant  to  be  answered  by  all  of  the  physicians  and  so  it  is 
necessary  to  follow  the  branching  pattern  in  the  questionnaire.  Finally, 
some  of  the  detailed  financial  questions  probably  require  that  the  physician 
consult  records  or     other  sources  of  information  in  order  to  provide  a  fully 
accurate  response. 

The  complexity  of  the  questions--mainly  caused  by  the  need  to  define 
and  explain  categories  of  cost  data--argues  against  using  a  mailed 
questionnaire.  Experience  from,  other  surveys  suggests  that  the  respondents 
would  generally  lack  the  motivation  to  work  through  the  questionnaire.  The 
complexity  of  the  instrument  would  also  argue  against  using  the  telephone 
except  for  the  fact  that  most  of  the  responses  required  are  simple  yes-no 
answers  or    dollar  values  and  so  there  is     less  of  a  problem  with  keeping 
the  question  and  the  response  categories  in  mind  than  is  usually  the  case 
in  telephone  sur'^eys. 
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The  use  of  screening  questions  argues  in  favor  of  either  the  telephone 
or  personal  interviews.  The  presence  of  more  questions  than  any  respondent 
is  expected  to  answer  makes  a  mailed  questionnaire  seem  even  more  formidable. 
In  addition,   there  is  the  need  to  print  and  explain  the  screening  instructions^ 
During  personal  or  telephone  interviews,   the  alternative  paths  through  the 
questionnaire  are  transparent  to  the  respondent. 

The  need  for  record  consultation  is  a  problem  that  points  to  the  need 
for  advance  notice  and  call  backs,  but  does  not  indicate  a  relative  advantage 
for  any  of  the  three  survey  methods.     Dillman  comments  on  the  relative  capacity 
of    each  of  the  three  methods  for  handling  this  problem: 

Each  method  has  certain  difficulties  associated  with  respondents 
obtaining  needed  consultation, ...  Consultation  usually  requires  interest 
and  motivation,   factors  that  provide  great  difficulty  for  users  of  mail 
questionnaires,  ,,, 

The  situational  constraints  under  which  telephone  interviews  are 
usually  conducted  are  not  generally  conducive  to  getting  information 
from  sources  other  than  the  respondent.   People  ...   are  less  likely  to 
be  willing  to  check  records,  consult  other  persons  to  find  the  needed 
information,  and  so  on.  ... 

The  difficulty  encountered  in  face-to-face  interviews  is  that 
the  person  who  has  the  necessary  knowledge  to  answer  certain  questions 
may  not  be  available  at  the  time  of  the  interview.    In  the  case  of 
written  records,   the  inteir^/iewee  may  not  know  where  to  find  them.  The 
only  alternative  in  some  cases  is  the  costly  rescheduling  of  a  later 
interview  for  a  time  when  the  most  knowledgable  household  member  is 
available.    . . , 

Thus  there  is  little  basis  for  recommending  one  method  over  another 
with  respect  to  the  ability  to  secure  accurate  data  when  consultation  is 
needed . 

p  65 

There  is  less  experience  and  less  research  available  on  the  question 
of  methods  bias     in  physician  responses,  particularly  for  the  kinds  of 
data  we  are  concerned  with  here,  Colombotos   (1969)  did  a  study  comparing 
physician  responses  to  a  telephone  ipten/iew  and  a  personal  inter-v/iew  which 
used    exactly  the  same  questionnaire.  He  reports  thit  there  were  no  differences 
in  physician  responses  to  the  following  kinds  of  questions:   personal  data 
(such  as  age,   religion  or  income);  attitudes  toward  government  participation 
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in  medical  care;  or,  other  attitudes  on  political  and  health  care  issues. 

He  did  find  differences  on  what  he  called  "social  desirability"  items  such 

as  reports  of  the  number  of  articles  published,  the  number  of  journals 

read  and  the  altruism  of  one's  motives  for  going  into  medical  practice.  ^ 

The  direction  of  the  bias  was  for  more  socially  desirable  responses  to  be  reported 

in  the  personal     interview.    In  a  subsequent  discussion  of  this  study 

Colombotos  reported  that  the  bias  in  the  report  cf  journal  articles  and 

publications  was  eliminated  when  the  interviewer  asked  the  physician  to 

list  exactly  which  articles  and  which  journals  were  included  in  the  response 

to  the  question.  Using  this  approach,   the  personal  inten/iew  and  the  phone 

interv/iew  produced  identical  responses   (Fowler,  1977). 

Other  studies  have  compared  the  three  data  collection  methods  but  not 
for  physician  samples.  Hochstim  (1967)  compared  the  three  methods  in  a 
survey  of  the  adults  in  a  metropolitan  area.     The  differences  he  observed 
between  the  three  methods,  using  the  same  questionnaire  were:    (a)   the  mail 
method  produced  fewer  codeable  responses  for  the  occupation  question,  but 
not  for  other  questions  requiring  detailed  instructions;    (b)  people  overrated 
their  level  of  personal  health  in  the  phone  and  personal  interviews;  and 
(c)  people  underreported  drinking  and  discussing  certain  medical  problems 
with  the  spouse  in  the  telephone  and  personal  inter^^iews.     Other  background, 
income  and  opinion  measures  showed  no  difference  by  method  of  data  collection, 

Wiseman  (1972)  in  a  study  of  adults  in  a  metropolitan  area  found 
no  difference  between  the  three  methods  in  reports  of  marital  status,  age, 
income,  occupation  and  religion  but  did  find  differences  in  responses  to 
certain  opinion  measures  on  topics  which  were,  at  the  time,  controversial. 

Finally,   Kahn  and  Groves   (1979)  report  a  great  many  subtle  differences 
betxveen  phone  and  personal  interview  responses.  Their  analysis  is  based  on 
sur-v'eys  of  the  adult  national  population.   They  find  differences   in  the 
n-umber  of  words  and  items  chosen  in  open-ended  questions,   in  the  response 
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alternatives  typically  chosen  for  attitude  questions , and ,  most  importantly 

for  our  purposes,   they  report  that  a  higher  percentage  of  people  are 

unwilling  to  provide  detailed  personal  financial  data  on  the  phone.  Specifically, 

there  were  more  refusals  to  discuss  one's  tax  return  and  one's  income  on 

the  phone  than  in  the  personal  interview.    Interestingly,   the  distribution 

of  income  and  the  average  tax  return  was  the  same  for  those  who  were 

willing  to  report  these  data  and  so  the  findings  of  the  findings  of  this 

study  are  completely  consistent  with  the  previous  studies.  The  difference 

is  that  fewer  people  were  willing  to  divulge  the  information  on  the 

phone.  This  is  a  fairly  serious  problem  for  a  survey  like  the  Physician's 

Practices  Cost  Survey,  and  so  we  will  devote  a  great  deal  of  attention  in 

the  next  section  of  this  paper  to  the  implications  of  this  result  for 

the  quality  of  the  data  provided  by  the  physicians  regarding  their  costs 

and  income. 

Generally  speaking,  when  we  consider  the  complexity  of  the  data 
to  be  collected,  there  are  clearly  reservations  about  the  efficacy  of  a  mail 
survey  for  the  problem  at  hand.  The  questions  in  the  Cost  Survey  seem  to  have 
little  to  do  with  the  social  desirability  items  which  are  known  to  be  biased 
in  phone  and  personal  interviews.     Possible  problems  with  the  income 
questions  indicate  a  preference     for  personal  interviews.  We  will  return  to 
this  topic  at  great  length  in  the  next  section. 

Costs  Given  the  information  we  have  already  presented  about  the 
difference  in  the  field  period  required  for  each  type  of  survey,  the 
ease  of  contact  with  telephone  interviewing,   and  the  ease  in  following  up 
cases  who  break  appointments  or  interrupt  the  interview,   there  should 
be  very  little  surprise  with  our  estimate  that  a  personal  interview 
using  the  same    design  specifications  would  increase  the  field  costs 
for  this  survey  by  30  to  50  percent.  The  question  is  would  a  mail  survey 
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be  cheaper?  We  believe  not.    It  is  most  likely  the  case  that  a  mail  survey 
of  physician  costs  and  practices  would  repeat  the  experience    of  the 
validation  survey  described  by  Andersen  and  Anderson  (1967). 
The  survey  was  initially  designed  as  a  mail  survey  but,  because  of  low 
response  and  incomplete  data,   it  was     necessary  to  switch  to  a  phone 
design  in  the  middle  of  the  field  period--adding  -o  the  cost  of  the  study. 


Summary:  Which  is  Better?  Figure  1  is  a  reproduction  of  the  summary 
table  from  Dillman's  book  which  highlights  the  strengths  and  weaknesses 
of  each  survey  method.  We  have  highlighted  the  areas  of  comparison  which 
are  most  relevant  for  the  Cost  Survey,   In  general , we  believe  the  conclusion 
from  this  summary  is  consistent  with  the  conclusion  of  the  first  national 
Conference  on  Health  Survey  Research  Methods  regarding  the  relative  efficacy 
of  personal  and  telephone  interviewing  for  collecting  medical  data: 

Advantages  and  disadvantages  of  telephone  interviews  were  discussed. 
The  advantages  presented  a  decided  edge  over  the  disadvantages.  The  major 
disadvantages  encompassed  sample  problems  and   (sample)   frame  development. 
Various  strategies  designed  to  reduce  errors  due  to  these  sampling  problems 
were  presented.    It  was  the  consensus  that,  with  the  proper  (sampling) 
strategy,   telephone  interviews  can  be  efficient.  There  are  many  advantages 
of  being  able  to  use  telephone  inters/lews  in  health  surveys.  Among  the 
advantages  discussed  were  the  following: 

(1)  One  can  assess  interviewer  performance  through  monitoring. 

(2)  One  can  interview  in  areas  where  interviewers  would  hesitate  to 
to  go,   particularly  during  the  evening  when  one  may  want  to 
reach  working  members  of  the  household. 

(3)  Health  professionals  are  more  likely  to  participate  in  telephone 
surveys  . 

(4)  Interviev/er  restrictions   (mobility,   transportation)  are  substantial 
lessened, 

(5)  Field  costs  are  reduced. 


from:   "Sumjnary  and  Conclusions",  Leo  G.  Reeder 

Advances  in  Health  Survey  Research  Methods:   Proceedings  of  a 
National  Invitational  Conference  Sponsored  jointly  by  the 
National  Center  for  Health  Services  Research  and  the  National 
Center  for  Health  Statistics 

No.    (HK-i)  77-3154  (1977) 
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II 

Sur\'ev  Measures  of  Personal  Finances 
 ^  

Almost  every  survey  seeks  to  obtain  some  measure  of  personal  or  family 
financial  status.   Because  of  this,   there  has  been  a  great  deal  of  concern 
about  the  ability  of  survey  organizations,   including  the  Census,   to  obtain 
accurate  financial  reports  in  interviews  or  mailed  questionnaires.  Unfortunately, 
the  am.ount  of  research  into  the  accuracy  of  financial  reports  is  well  belov;  v/hat 
we  might  expect  it  to  be  given  the     level  of  concern  about  the  topic.    In  this 
section  we  will  discuss  the  results  from  the  studies  which  are  most  helpful 
for  understanding  the  problem^s  with  obtaining  financial  data  from  physicians 
and  other  professionals.    (There  are  several  reports  on  the  accuracy  of 
reports  of  xvelfare  payments  received,   food  stamps,   and  so  on,  which  would 
be  somewhat  beside  the  point  of  the  present  concern.) 

The  three  threats  to  the  validity  of  personal  financial  reports  are: 
(1)  people  may     refuse  to  answer  the  questions;    (2)  people  who  are  willing 
to  answer  the  question  might  not  accurately  remember  their  income  or  financial 
status;   and,  (3)  people  may  purposely  distort  their  response  to  the  question. 
Each  of  these  responses  threatens  to  bias  the  estim.ated  level  of  financial 
well-being  in  the  sample.    In  addition,   it  may  be  the  case  that  certain  of  these 
response  patterns  are  more  prevalent  in  particular  groups  in  the  sample 
(e.g.  very  wealthy  people  are  m.ore  likely  to  refuse  to  state  their  income;  older 
people  forget  to  add  in  certain  sources  of  income)   in  which  case  there  is  a 
differential  bias  which  will  contaminate  certain  group  comparisons  as  well  as 
overall  estimates  of  mean  or  m.edian  income. 

Our  review  of  the  literature  finds  that  under  certain  circumstances 
each  of  the  three  threats  to  validity  operates  in  the  reporting  of 
financial  data.   For  physician  sur^/eys ,  however,   the/main  threat  to  validity 
arises  from  the  refusal  problem.,   racher  chan  forgetting  or  misrepresent- 

ation. Ve  find  a  little  evidence  on  the  question  of  the  relative  efficiency 
of  mail  questionnaire  vs.   phone  inter\" iewing  vs.   personal  interviewing 
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as  a  method  for  collecting  financial  data.     What  evidence  there  is  suggests 
that  if  the  goal  of  a  survey  is  to  obtain  detailed,  highly  precise  estimates 
of  financial  data  then  there  is  a  clear  preference  for  personal  interviews. 
If  one  is  willing  to  tolerate  a  coarser  grain  to  the  data  then  mail 
questionnaires  can  produce  more  accurate  responses  i^  the  survey  cooperation 
rates  are  sufficiently  high.  For  sur\'eys  of  physicians,  however,  the  data 
show  that  mail  and  telephone  inten/iews  are  about  equally  accurate.  There  is 
a  loss  in  efficiency  in  estimating  income  because  of  the  physicians  who 
refuse  to  answer  the  question,  but  this  does  not  appear  to  bias  sample  estimates. 

Refusing  to  Answer  There  is  not  much  systematic  research  showing 
why  people  refuse  the  income  questions.  A  quick  analysis  of  the  NORC 
General  Social  Survey  shows  that  in  a  national  cross  sectional  probability 
sample  of  adults  we  can  expect  that  about  6  percent  will  refuse,  say 
they  "don't  know"  or  somehow  avoid  the  personal  incom.e  question.  This 
percentage  is  the  same     for  those  in  blue  collar  and  those  in  white 
collar  jobs.    (Interestingly,  of  the  12  doctors  and  dentists  sampled  in  the 
last  5  years  of  the  General  Social  Survey,  none  refused  the  personal 
income  question  .)  A  more  detailed  multivariate  analysis  shows  that 
refusing  the  personal  income  question  is  not  related  to:   the  prestige  of 
one's  job;   region;   education;   satisfaction  with  one's  job;  marital  status; 
or  urban  vs.   suburban  vs.   rural  place  of  residence. 

On  the  other  hand,   refusing  the  income  question  is  related  to 
several  interesting  variables  and  from  these  we  can  begin  to  understand 
(or  at  least  quantify)  our  doubts  about  the  incompleteness  of  the  data. 
In  a  multivariate  analysis,  we  find  the  following  variables  to  show 
significant  net   (i.e.   partial)  contributions: 

(a)  Race,  blacks  are  3  percent  less  likely  to  report  than  whites 

(b)  Age,  people  over  35  are  3  percent  less  likely  to  report  than  those  younger 

(c)  Sex,  women  are  2  percent  less  likely  to  report  their  personal  income  than  men 

(d)  Intelligence,   people  who  score  below  the  median  on  a  vocabulary  test 

are  2  percent  less  likely  to  rq)ort  than  those  who  score  above 
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(e)  "Conservatism",  those  who  describe  themselves  as  Extremely  Conservative 
or  Conservative  (as  opposed  to  Slightly  Conservative,  Middle  of 
the  Road  or  Liberal)  are  more  likely  to  refuse 

These  findings,  particularly  the  effect  of  conservatism  suggest  that 
refusal     to  answer  might  be  a  result  of  anti-prying  and  possibly  "anti  big 
government"  kinds  of  attitudes.   Several  further  analyses  did  not  support  this 
exact  interpretation,  however.  The  percent  refusing  fluctuates  randomly  between 
1973  and  1978,   a  period  when  anti-governm.ent  attitudes  and  concerns  for 
personal  privacy  have  supposedly  been  on  the  increase.   Other  measures  of 
confidence  in  governm.ent  and  alienation  from  society  show  no  relation  to 
refusing  the  income  question. 

What  is  even  more  difficult  to  explain  is  that  over  one  third  of  those 
who  did  not  supply  personal  income  data  were,   in  fact,  willing  to  answer  the 
family  income  question  V7hich  used  the  same  response  categories. 

In  terms  of  national  samples  of  the  adult  population,  the  results  suggest 
that  the     effect  of  the  refusal  problem  is  to  overstate  the  level  of  income 
in  the  population.  The  people  who  do  not  answer  the  question  appear  to  be  those 
who  are  more  likely  co  have  lower  paying  jobs--v/omen,  blacks,   less  intelligent 
people.   On  the  other  hand,   the  age  differences  suggest  some  tempering  of  this 
conclusion  is  in  order.  All  things  equal,  people  in  the  middle  age  ranges 
probably  make  more  than  those  younger  and  so  the  higher  rate  of  refusal 
in  this  group  ought  to  cause  some  understatement  of  the  level  of  income  in 
the  population. 

Obviously  we  cannot  generalize  these  results  directly  to  the 

physicians  in  our  sam.ples.  We  do  recommend,  however,   taking  the  results  seriously 

as  possibly  explaining  the  physician  refusals.   The  oercent  of  physicians 

V 

refusing  to  provide  income  data  in  the  first  Physician's  Practice  Cost  Survey 
is      shown  in  Table  1. 
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Table  1 

Percent  of  Physicians  Refusing  Gross  and  Net  Income  Questions 

Office  Based  Hospital  Based 

(15  Specialties)  (3  Specialties) 

N=3482  N=543 

Gross  Income                13.7  18.4 

Net   Income                   16.4  19.7 
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The  percentages  are  uncomfortably  high  and  therefore  we  would  recommend  that 
some  analysis  of  the  correlates  of  refusing  the  income  question  be  done 
so  that  one  can  assess  the  likelihood  and  probable  direction  of  bias. 

Direction  of  Bias  Among  Those  Providing  Data  The  second  and  third  threats 
to  the  validity  of  personal  financial  reports  were:   forgetting  to  include 
certain  assets   (or  liabilities)   in  one's  report;   and,  purposely  distorting 
one's  answer.  These  threats  are  discussed  together  in  this  section. 

Most  of  the  studies  which  have  used  external  records  to  validate  income 
reports  find  that     people  who  answer  the  question  usually  provide  accurate 
information,  Keating,  Paterson  and  Stone  (1950)  compared  weekly  wages  obtained 
in  an  interview  and  employers'  wage  reports;   they  found  a  Pearson  correlation 
coefficient  equal  to  .90  for  a  male  sample  and  a  coefficient  of  .93  for  a 
female  sample.  Mosel  and  Cozan  (1952),   in  a  study  of  employment  applications, 
found  similarly  high  correlations  betv/een  job  applicants'   reports  and  employers 
records  of  past  wages.   Studies  matching  data  from  other  sources  with  the  1950 
Census  statistics   (National  Bureau  of  Economic  Research,   1958)   found  that 
differences  in  m.edian  earnings,   as  presented  to  Census  enumerators  and  as 
secured  from  other  sources,  were  relatively  minor.   Finally,   Borus   (1966)   ,  in 
a  sur^/ey  of  participants  in  government  retraining  programs,   found  that  there 
was  very  little  overall  bias  in  the  reports  of  earnings  but  that  there  were 
systematic  differences  in  the  accuracy  of  reports.   Those  who  were  better 
educated  and/or  who  had  higher  total  earnings  were  more  accurate  in  their 
reports . 
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We  might  question  the  appropriateness  of  each  of  these  studies  for 

generalizing    to  our  own  sample  of  physicians.   Of  the  validation  studies,  only 

the  NBER  income  analysis  involved  a  middle  or  upper  income  population  and 

this  study  dealt  with    aggregate  statistics--individual  reports  of  income 

were  not  linked  to  individual  records.   It  does  not  appear  to  be  the  case 

that  studies  have  been  conducted  to  validate  the  income  rg)orts  of  high 

income  respondents.  There  have,  however,  been  studies  of  the  accuracy  of 

reports  of  other  assets  such  as  savings  accounts.  These  studies  involve  a 

sample  of  respondents  which  is  much  closer  in  social  status  to  the  physician 

samples  and  so  we  turn  to  a  brief  discussion  of  the  findings. 

In  a  validation  of  savings  account  reports  in  the  Netherlands,  Maynes 

(1965)  advances  the  hypothesis  that  respondents  bias  their  reports  to 

conform  with  their  notion  of  the  holdings  of  the  "average  person"  of  their 

social  status.  That  is,  people  with  extremely  high  balances  tended  to  underreport 

their  holdings  and  people  with  belov/  average  holdings  tended  to  inflate  their 

worth.   Ferber  (1956)    found  the  same  pattern  but  s-howed  that  the  reason  for  the 

effect  was  not  misreporting .  Rather,  the    "average  person"  effect  is  largely 

explained  by  including  people  who  falsely  report  "no  assets"   (i.e.   those  who 

refuse  the  question)  in  the  distribution  of  responses.    In  general,  Ferber' s 

conclusions  point  to  item  refusal  and  survey  nonresponse  as  the  major  sources 

of  bias  in  the  estimates  of  assets: 

Overall,   these  time-deposit  validation  studies  point  to  the 
existence  of  substantial  errors  in  such  data  when  obtained  by  consum.er 
surveys . Thus ,  we  find  that: 

(1)  Sample  averages  tend  to  understate  substantially  the  true 
population  figures. 

(2)  As  a  general  rule,   the  principal  cause  of  these  underestimates 
is  the  nonreporc ing   (i.e.   item  refusal)  of  respondents.  A  second 
major  factor  appears  to  be  underestimates  of  the  holdings  of 

the  nonrespondents   (i.e.   of  those  refusing  the  sur^vey.) 

(3)  Response  errors   (i.e.  bias  among  cnose  reporting)  are  generally 
small.  As  was  true  in  the  case  of  debt,   there  is  some  tendency 
toward  overstatement,  but  this  tendency  is  slight. 
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(4)  A  finding  that  runs  counter  to  the  corresponding  one  for  debt 

is  the  overstatement  of  the  frequency  of  small  savings  accounts. 
There  is  also  some  tendency  for  the  frequency  of  large  savings 
accounts  to  be  understated,  particularly  if  non-reporters  are 
included  in  the  size  distributions.  This  tendency  toward  the 
so-called  average  man  effect,.,has  been  found  to  exist  in  a  ^ 
number  of  other  instances. 

(5)  The  tenuous  generalization  in  the  case  of  debt  that  nonrespondents 
hold  more  than  respondents  do  receives  strong  support  in  the  case 
of  savings  accounts.   In  some  of  these  studies,   the  average 
balances  of  the  nonrespondents  were  several  times  as  large  as  those 
of  the  respondents. 

pp  123-124  Ferber,19&6 

Data  Quality  and  Type  of  Interview  Most  research  shows  that  personal 
financial  data  is  the  hardest  type  of  data  to  collect  from  survey  respondents. 
In  a  study  of  methods  of  collecting  data  on  threatening  topics,  Bradburn 
and  Sudman  (1979)   find  that  the  only  kinds  of  questions  which  make  people 
more  uneasy,  which  interviewers  consider  harder  to  ask  and  v;hich  produce 
higher  levels  of  nonresponse  are  detailed  questions  about  personal 
drinking  or  sexual  habits.   Some  of  the  illustrative  data  are  shown  in  Table  2. 

The  question  we  shall  concentrate  on  now  is:   "Is  income  data  even  haraar 
to      collect      on  the  telephone  than  in  personal  interviews?"   In  terms  of 

the  percent  v/ho  are  uneasy  about  providing  the  information,   the  answer 
appears  to  be  "yes."  In  terms  of  the  percent  who  refuse  the  question^ the 
answer  is  qualif ied--it  appears  to  be  the  case  that  one  cannot  collect 
the  same  level  of  detail  on  the  phone  as  in  personal  interviews,  but  that 
if  one  settles  for  coarser  grained  data,   there  is  not  a  drop  off  in  item 
response.   In  terms  of  the  accuracy  of  the  information  provided,   the  studies 
which  have  been  done  report  no  difference  in  the  level  of  income  reported 
for  sun/eys  done  by  phone,  mail  and  by  personal  interviews. 

In  a  study  comparing  telephone  and  personal   intervievrs,   Kahn  and 
Groves   (1979)   report  chat  incom.e  is  the  most  sens^itive  question  on  the 
survey  and  that  the  effect  is  aggravated  for  phone  surveys.  The  data  are 
shown  in  Table  3. 
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Table  2 


Ratings  of  Uneasiness  and  Difficulty  of  Question  Topics 


Question  Topics 


Respondent  Rating 
of  Percent  of 
People  Made 
"Very  Uneasy" 
By  a  Topic   


Question  Caused 
Difficulty 
( Interviewer 
Rating) 


Percent 
Not 

Responding 
To  AcuPal 
Question'' 


Sports  Activities  1 

Leisure  Time  2 

Social  Activities  2 

Occupation  3 

Education  3 

Happiness  and  Well-Being  4 

Drinking  Beer,  Wine,  Liquor  10 


Gambling  with  Friends 


10 


0 
0 
4 
3 
2 
6 
10 
3 


.1 

.2 
.8 
.1 
.3 
,3 
.1 
.2 


Income 


12 


4.8 


Getting  Drunk 
Sexual  Intercourse 
Masturbation 


29 
42 
56 


9 
27 
29 


2.3 

6.0 

6.7 


Source:   Bradburn  and  Sudman  (1979) 
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Table  3 

Percent  Reporting  They  Were  Uneasy  With  Particular  Questions 


Percent  uneasy  with  Type  of  Interview 


questions  regarding,.. 

Telephone 

Personal 

One's  job 

3 

2 

One's  health 

3 

2 

Voting  behavior 

9 

8 

Racial  attitudes 

9 

9 

Political  opinions 

12 

9 

Income  tax  return 

14 

9 

Income 

28 

15 

Source;   Kahn  and  Groves,  1979 


/ 
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One  of  the  difficulties        with  comparing  the  results  of  the  different 
studies  of  the  sensitivity  of  the  income  question  is  that  different  surveys 
ask  the  question  in  different  ways.     I<rnen  personal  income  is  asked  as  a 
dollar  amount,     the  percent  refusing  increases  whether  one  interviews  on  the 
phone  or  in  person.   In  the  Kahn  and  Groves  study,   income  was  asked  in  the 
following  way: 

"Finally,  hox^/  much  did  you  earn  from  your  job  last  year,  in  1975, 
before  taxes  and  other  deductions  were  made?  (Again,  just  to  the 
nearest  thousand  dollars.)" 

When  this  version  was  used  in  the  personal  interview  8.3  percent  refused  to 

answer  the  question.  This  is  higher  than  the  6  percent  nonresponse  from  other 

household  surveys  and  the  difference  is  attributable  to  the  difference  in 

question  wording.   Kahn  and  Groves  found  that  the  same  question  asked  over  the 

phone  produced  a  24, 1  percent  nonresponse  rate--even  higher  than  the 

nonresponse  rates  for  the  physician  samples  reported  in  Table  1,    If  the 

respondent  refused  to  answer  in  the  telephone  interview  for  the  Kahn  and 

Groves  study,   then  there  was  a  follow  up  question: 

(IF  RESPONDENT  REFUSES  TO  GIVE  DOLLAR  FIGURE) 

"We  don't  need  the  exact  dollar  figure;  would  you  tell  tell  me  which 
of  these  three  broad  categories  it  falls  in  . , . 

1.  less  than  $7 , 500 

2.  between  $7,500  and  $15,000 

3.  more  than  $15,000 

Many        respondents  were  willing  to  answer  the  less  detailed  question, 
lowering  the  overall  nonresponse  rate  to  8.7  percent. 

The  implication  from  this  is  that  one  does  not  necessarily  suffer  greater 
nonresponse  to  the  incom.e  question  on  the  phone  if  one  is  willing  to  ask  the 
question  in  very  general  categories.  The  incom.e  question  the  Physicians' 

Practices  Cosl  Survey  do-      not  ask  for  an  exact  dollar  amount.  Instead 
there  are  several  detailed  categories  for  response^  The  question  for 
emplo^/ee,  unincorporated  sole  or  unincorporated  group  practitioners  was: 
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"In  xvhich  of  these  broad  groups  was  your  o\m  gross  income  from  medical 
practice  in  1976  before  practice  deductions  and  taxes.  Was  it  ... 

1.  less  than  $40,000 

2.  $40,000  to  less  than  $50,000 

3.  $50,000  to  less  than  $60,000 

4.  $60,000  to  less  than  $70,000 

5.  $70,000  to  less  than  $80,000 

6.  $80,000  to  less  than  $90,000 

7.  $90,000  to  less  than  $100,000 

8.  $100,000  to  less  than  $150,000 

9.  $150,000  to  less  than  $200,000 

10.   $200,000  or  over?    ...   About  how  much  was  that?  $  


The  gross  income  question  for  partnership,   incorporated  solo  or  incorporated 


group  practitioners  was: 


"In  vzhich  of  these  broad  groups  was  the  gross  income  of  the  partnership 
or  corporation  during  1976  before  practice  deductions  and  taxes?  Was  it 

1.  $50,000  to   less  than  $75,000 

2.  $75,000  to  less  than  $100,000 

3.  $100,000  tc  less  than  $125,000 

4.  $125,000  to  less  than  $150,000 

5.  $150,000  to  less  than  $200,000 

6.  $200,000  to  less  than  $250,000 

7.  $250,000  to  less  than  $300,000 

8.  $300,000  to  less  than  $400,000 

9.  $400,000  to  less  than  $500,000 

10,  $500,000  or  over?   ...  About  how  much  was  that?  $  

Finally,  the  net  income  question  for  all  physicians  also  used  several 


categories : 


"In  which  of  these  broad  groups  was  your  own  net  income  from  the  practice 
of  medicine  in  1976  after  practice  deduct  ions  and  before  taxes?Was  it,,. 

1.  less  than  $20,000 

2.  $20,000  to  less  than  $30,000 

3.  $30,000  to  less  than  $40,000 

4.  $40,000  to  less  than  550,000 

5.  $50,000  to  less  than  $60,000 

6.  $60,000  to  less  than  $70,000 

7.  $70,000  to  less  than  $80,000 

8.  580,000  to  less  than  $90,000 

9.  390,000  to  less  than  $100,000 

10,   $100,000  or  over?   ...  About  how  much  was  that?  $  
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One  implication  from  the  Kahn  and  Groves  study  is  that  there  might  have  been 
fewer  refusals  if  the  income  questions  had  used     fewer  and  broader  categories, 

V\Jhat  about  the  accuracy  of  the  data  collected'^'  Does  the  mode  of 
interviewing  bias  the  responses  obtained  from  those  who  are  willing  to  answer 
the  income  questions?  The  studies  done  so  far  say  no.   Jordan,  Marcus  and 
Reeder,   in  a  health  survey  of  the  general  population  in  a  metropolitan  area 
find  that  there  are  more  incom.e  refusals  in  the  telephone  survey  but  that 
the  distribution  of  responses  is  the  same  for  those  willing  to  report  income 
in  a  personal  and  in  a  telephone  sur^/ey  (1979).   Colombotos   (1969)  in  an 
experiment  comparing  physicians'   responses  to  personal  and  telephone  surveys 
finds  no  difference  in  the  distribution  of  reported  income  between  the  txw 
methods.   Ferber  (1966)     argues  that  mail  questionnaires  are  preferable  to 
personal  interviews     if    detailed  information  is  to  be  collected  about 
several  sources  of  income  (earnings  as  well  as  other  assets)  and  if  the 
response  rate  to  the  mail  survey  can  be  kept  sufficiently  high.    In  a  test 
of  the  comparability  of  mail  and  telephone  surveys,   Goodman  and  Jensen  (1979) 
compare  the  mean  income  from  the  AMA's  Periodic  Sur^/ey  of  Physicians  with 
the  Physicians'   Practices  Cost  Survey  for     five  specialty  groups.  The  mail 
questionnaire  produced  a  slightly  higher  mean  reported  income  for  General/ 
Family  Practitioners,   Internal  Medicide  Specialists  and  Surgeons  and  a  slightly 
lower  estimate  for  Pediatricians  and  Ob-Gyn  Specialists.   Combining  the 
groups,   the  mail  sur^/ey  produced  an  aggregate    mean  income  that  was  about 
$3000  or  6  percent  higher  than  that  reported  in  the  telephone  survey.  The 
question  wording  for  the  m.ail  survey  asked  for  a  dollar  am.ount .  The 
question  was; 

"T,vhat  was  your  1975  i:^IDIVIDUiiJ:.  NET  INCOMi:  BEFOBi:  TAICES  from  medical 
practice  to  the  nearest  $1,000?"   (Gaffney,  19^9) 
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The  asking  for  a  dollar  value  may  be  one  of  the  explanations  for  the  low 
response  rate    to  the  A}IA  survey.    Item  nonresponse  rates  are  not  given,  but 
Goodman  and  Jensen  report  that  the  survey  refusal  rate  is  near  50  percent. 

The  conclusion  to  this  section  is  that  there  appear  to  be  few  method 
differences  in  reported  incom.e  for  those  x-/illing  to  ansv7er  the  question  but 
that,   for  the  same  level  of  detail,   item  response  rates  are  highest  in 
personal  interviews  and  lowest  in  m.ail  questionnaires.  We  should  note  that 
the  finding  of  little  difference  between  survey  methods  in  the  level  of  reported 
income  does  not  mean  that  the  level  of  reported  income  is  unbiased.  What  we 
have  learned  is  that  if  there  is  a  bias,   it  is  approximately  the  same 
under  each  method  of  data  collection.  The  question  of  bias  was  addressed 
in  the  previous  section  where  scattered  studies  of  very  different  populations 
show  that  people  who  are  willing  to  report  their  income  and  assets  generally 
do     so  accurately. 

Can  We  Improve  Incom.e  H.eporting?     By  and  large,   this  question  means: 
can  we  obtain  higher  response  rates  to  the  income  question,   since  nonresponse 
is  the  main  source  of  imprecision.  At  the  beginning  of  this  section  we 
examined  some  of  the  correlates  of  income  nonresponse  in  the  national  adult 
population.   These  findings  did  not  take  us  too  far  in  developing  a  theory  of 
nonresponse.   In  their  work  on  data  collection  for  threatening  topics,  Bradburn 
and  Sudman  found  two  factors  that  correlated  strongly  v/ith     the  quality  of  income 
reports;  the  amount  of  confidentiality  promised  by  the  interviewer;  and, 

the  interviewer's  expectation  that  income  would  be  over  or  underreported . 
In  an  experimental  comparison,   7  percent  of  those  who  had  been  given  an 
absolute  assurance  of  confidentiality  were  unwilling  to  report  their  incomie, 
whereas  the  rate  of  nonresponse  among  those  to  whom  no  m.ention  was  m.ade  of 
confidentiality  was  13  percent.   Furthermore,   the  re'fusal  rate  was  33  percent 
am.ong  those  who  refused  to  sign  an  informed  consent  form  as  opposed  to  10 
percent  among  those  who  did.   Each  of  these  findings  points  to  the  trem.endous 
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importance  of  trust  and  belief  that  the  sur^/ey  is  truly  confidential  in 
the  decision  to  report  one's  income, 

Bradburn  and  Sudman  also  found  that  interviewer  who  believed  that  income 
was     generally  -  underreported  in  sur^/eys  in  fact  collected  income  data  that 
was,  on  average,   $2000  per  year  less  than  the  income  reports  collected  by 
interviewers  who  believed  that  income  was  accurately  reported  or  overreported. 
This  finding  points  to  the  importance  of  inter^/iewer  training  and  inter^/iewer 
expectations  in  collecting  income  data.   In  some  sense,  we  stack  the  cards 
against     us  by  stating  that  incom.e  data  is  hard  to  collect  since  this  feeds 
the  expectation  that  income  data  will  not  be  collected  or  inaccurately  collected 
which,   in  turn,  can  become  a  self-fulfilling  prophecy.  This  line  of 
reasoning  is  consistent  with  the  reports  by  Kahn  and  Groves   (1979)  and  by 
Jordan,  Marcus  and  Reeder  (1979)  that  the  refusal  rate  for  income  declined 
as  interviewers  gained  more  experience  with  the  survey. 

In  terms  of  improving  response  rates  for  the  income  question,   it  appears 
that  we  can  make  progress  by  making  m.ore  reassuring  statements  about 
confidentiality,  by  training  interviev^ers  more  carefully  and  by  advising 
inter\' iev7ers  that  it  is,   in  fact,  possible  to  collect  accurate  income  data 
from  a  very  high  percentage  of  the  sample  respondents,  Ferber  (1966)  also 
reports  a  provocative  finding  that  there  were  fewer  incom.e  nonresponses  when 
the  incomie  and  assets  questions  were  preceded  by  questions  about  indebtedness. 
Apparently,   the  discussinn  of  debts  broached  the  general  topic  of  financial 
status  in  a  way  that  was  threatening  to  fewer  respondents. 

In  addition,   there  is  evidence  that  som.e  ways  of  asking  the  incom.e 
question  provide  m.ore  accurate  responses   (for  those  v/ho  agree  to  respond)  than 
do  ochers.   I'fnen  total  income  from  several  different  sources  is  requested,  then 
there  is  experimental  evidence  that  it  is  preferable  to  ask  one  or  more 
detailed  questions  about  each  source  of  incomie  (Borus ,   1970;  Herriot,  1977). 
Even  when  the  goal  is  to  obtain  income  from  one  source  using  response 
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categories   (rather  than  asking  for  dollar  values)   there  is  evidence  that 

it  is  better  to  start  in  the  middle  of  the  list  of  categories     and  branch 

up  or  down  rather  than  start  at  the  bottom  (as  in  the  Physicians'  Practices 

Costs  Survey)  or  at  the  top.   Specifically,  Locander  and  Burton  (1976)  have 

shoTO  in  a  study  of  an  American  metropolitan  area  that  an  income  question 

of  the  following  form  produces  less  biased  responses   (they  do  not  address 

the  question  of  refusal  rates): 

"I'Jhat  was  the  approximate  annual  income  for  all  members  of  your  family 
before  taxes  during  1974, . .would  it  be  $15,000  or  more  or  would  it  be 
less  than  that? 

(IF  $15,000  OR  MORE  ASK): 

Would  it  be  Under  $20,000? 
(IF  OVER  $20,000  ASK) : 

Would  it  be  Under  $25,000? 


(IE  LESS  THAN  $15,000  ASK): 

Would  it  be  over  $10,000? 
(IF  UNDER  $10,000  ASK): 

Would  it  be  over  $7,500? 

The  results  from  the  Locander  and  Burton  study  show  that  this  form  of  the 
incom.e  question  yields  a  more  even  distribution  of  responses  relative  to  the 
other  ways  of  asking  the  pre-coded  category  questions.  A  list  starting  at 
the  bottom  category  (as  in  the  Physicians'   Costs  Survey)   skrews  the  distribution 
slightly  toward  lower  incomes,  A  list  of  questions  starting  at  the  top 
category  skews  the  distribution  the  other  way.   It  appears  that  a  slightly 
better  distribution  of  income  for  the  Physicians'   Practices  Costs  Survey 
would  have  been  obtained  by  beginning  the  questioning  near  the  median 
income  for  all  physicians  and  then  branching  up  or  down  depending  on  the 
responses.   Interestingly,   this  is  one  possible  explanation  for  the 
differance  between  the  mail  and  telephone  3ur">/ey  ^es  tiraates  of  physicians' 
incomes  earlier. 
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An  Evaluation  of  the  Income  Data  From  the  Costs  Survey  To  summarize 

this  section,  we  have  several  comments  about  the  quality  of  the 

income  data  and  the  feasibility  of  obtaining    more  accurate  reports: 

1    The  income  measure  used  in  the  survey  is  a  relatively  accurate  measure 
for  those  responding.  The  accuracy  of  the  measure  could  possibly 
be  improved  by  using  an  alternate  version  of  the  question  described 
in  this  section.   Experimental  vork  on  this   topic        could  be  arranged 
at  a  fairly  low  cost. 

2.  Steps  could  be  taken  to  reduce  the  income  refusals  in  the  survey.  A 
statement  about  confidentiality  preceding  the  income  question  could 
increase  the    percent  responding.  A  short  form  of  the  income  question 
could  be  used  for  those  who  refuse  the  first  question.   Each  of  these 
steps  ought  to  increase  the  interviewer's  confidence  that  income 

can  be  obtained  in  the  survey,  which,   itself  ought  to  increase 
the  percent  responding. 

3.  If  the  m.ain  emphasis  of  the  survey  was  to  obtain  highly  precise 
estim.ates  of  physician  income,   then  we  would  recommend  further 
redesign  of  the  income  question  along  the  lines  of  the  Current 
Population  Survey  approach--detailed  questions  about  several 
subcategories  of  income  which  are  aggregated  for  an  overall 
measure.   Given  the  several  aims  of  the  survey  and  the  likely 
use  of  the  income  variable  (as  a  classification  of  physicians' 
relative  standings  for  tabulation  analysis)  we  believe  that 

the  design  and  responses  to  the  income  question  provide  data 
which  are  satisfactory  and  appropriate. 


4.   The  steps  required  to  actain  the  next  quantum  level  of  accuracy 
(i.e^.   steps  beyond  those  in  2)  are  either  prohibitively  expensive 
or  objectionable  for  privacy/confidentiality  reasons.    In  order  to 
achieve^  improvement  of  a  major  magnitude,   it  would  be  necessary  to 
audit  physicians'    financial  records  or  use  other  governmencal  source; 
of  data  such  as  tax  returns. 


5.  Even  with  the  cualif ications  stated  in  1,2  and  3  we  believe 
iz  is  important  to     acknov7ledge  thac  the  income  and  expense 
data  collected  in  the  Costs  Sur^/ey  are  the  besc  available  for 
any  national  sample  of  ohysicians. 

\ 
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III 

Missing  Data 

There  are  two  types  of  missing  data:   person   (or  household)  nonresponse; 
and,   item  nonresponse.   Every  survey,   regardless  of  the  data  collection  technique 
used  and  regardless  of  the  sponsoring  agency   (even  the  U.S.   Census)  will 
be  prone  to  a  certain  level  of  missing  data  of  both  kinds.  Statisticians 
and  survey  researchers  have  developed  procedures  for  working  with  the 
problems  presented  by  missing  data.  There  have  been  several  models  for 
follov7-up  and  subsampling  proposed  to  handle  the  problem  of  person  or  household 
nonresponse.  These  procedures  are  described  in  a  recent  review  article 
(Daniel, 1275) . 

We  have  been  specifically  asked  to  address  the  problem  of  item 
nonresponse.     VThen    data  are  missing  for  responses  to  specific  items  then 
there  is  the  risk  of  two  undesirable  outcomes.   First,   it   is  possible  that 
an    estimate  of  the  average  response   (mean,  percent,   etc)  based  on  the 
data  for  those  who  answered  the  question     is  a  biased  estim.ate  of  the  "true" 
average  in  the  sample.  This  would  be  the  case  if  V7ealthy  people  were  more  likely 
refuse  the  income  question.  Then,   the  estimate  of  the  level  of  personal  income 
in  the  responding  sample  is  lower  than  the  level  of  income  in  the  sample 
actually  drawn.  The  second  undesirable  outcome  results     from  the  decision 
to  analyze  the  correlations  betxveen  variables,   some  of  v/hich  have  missing 
observations  and  others  not.  What  is  one  to  do  with  the  case  where  there 
is  missing  data  for  one  variable  in  a  bivariate  correlation  analysis  but 
not  the  other?  Should  one  discard  the  obser'vat ion?  This   is  the  usual  solution, 
but  statisticians  have  noted  that  to  do  so  actually  makes   extremely  com.plex 
and  probably  unjustified  assumptions  about  the  distribution  of  obser^/ations 
in  the  sam.ple  (Hertel ,   1976)  and  will  result  in  a  biased  estimate  of  the 
correlation. 
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To  protect  against  the  biases  resulting  from  inappropriate  handling 
of  missing  item  responses,   the  U.S.   Bureau  of  the  Census  and  other  statisticians 
have  developed  routines  for  imput ing  observations.   The  imputation  routines 
essentially  ask:   "What  is  the  response  most  likely  to  be,   given  the  infor- 
mation that  is  available  about  the  individual  and  the  sample  from 
which  the  individual  obser-'/ation  is  drawn     (Bureau  of  the  Census,  1969)?" 
There  are  several  different  m.ethods  for  imputation  described  in 

a  recent  review  article  by  Hertel   (1976).     Some  of  the  routines  are  computation- 
ally more  expensive  than  others.  Unfortunately,   the  expensive  ones  are 
also  the  ones  x^hich  go  the  furthest  in  freeing  the  data  set  from  biased 
measures  for  means,  variances,  correlations,  and  the  like.  The  Census 
procedure    classifies  the  sample  into  several  relatively  homogeneous  groups 
(e.g.   age  groups  v;ithin  medical  specialty  classifications)   and  then  randomly 
substitutes  answers  from  those  who  have  responded  to  the  sur^/ey  questions 
into  the  data  records  for  those  who  did  not  answer  the  question  or  who,  for 
some  other  reason, have  a  missing  observation. 

It  can  be  mathematically  shown  that  imputation  routines  can  never 
succeed  in    predicting  the  true  scores  for  missing  observations  with  100 
percent  accuracy.   The  argument  that  is  advanced  in  favor  of  imputation 
is  that  one  can,   in  fact,  never  escape  the  need  for  imputation,  The  choice 
one  faces  is  to  either  ignore  the  missing  data  problem- -which  produces 
biases  in  the  statistics  estimated- -or ,   to  use  the  imputation  method  that 
meets  one's  needs,  given  the  budget.  The  imputation  cannot  recapture  the 
lost  data,  but  it  can  offer  better,   less  biased  estimates  for  the  statistics 
of  interest. 
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NORC  has  no  automatic  policy  on  imputation.   We  do  not  choose  a 
procedure  nor  do  we  provide  imputed  estimates  unless  these  steps  are 
specified  in  our  survey  contracts.   All   things  equal,  we  would  recommend 
the  im.putation  procedure  which  minimizes  the  expected  bias     from  the 
missing  obser^/ations .   Operationally,   this  usually  m.eans  the  Census 
procedure  or  other  advanced  m.ethods  based  on  m.ultiple  regression  models 
for  predicting  the  missing  observation  (Hertel,  1976). 
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